Add alternative setup instructions through Anaconda #139
Conversation
…-required-dependencies
|
|
||
| conda info --envs | ||
|
|
||
| #. Install Python requirements and VariantWorks package from source |
There was a problem hiding this comment.
Could not combine the pip installation step with theenvironment.yml file like so
-pip
- -r file:requirements.txt
because we don't know the user-defined environment location prior to running the conda env create command to set LDFLAGS & CPPFLAGS.
There was a problem hiding this comment.
Yes I think it's okay to keep them separated for now.
There was a problem hiding this comment.
Using explicit package names in the environment.yml file:
- bioconda::htslib
- pip:
- cyvcf2
also results in the following error when creating the environment:
also results in:
-Wno-discarded-qualifiers
htslib/bgzf.c:38:10: fatal error: zlib.h: No such file or directory
38 | #include <zlib.h>
| ^~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
| #. Some Python packages depends on the headers and shared libraries which were installed in the previous step, | ||
| therefore before executing `pip`, run the following command: | ||
|
|
||
| .. code-block:: bash |
There was a problem hiding this comment.
Instead of defining these env variables, I tried to pass to the pip install -r requirements.txt command the following arguments:
--global-option=build_ext --global-option="-I<ENVIRONMENT_PATH>/include/" --global-option="-L<ENVIRONMENT_PATH>/lib"
or even add these arguments to the relevant entries in the requirements.txt files:
biopython==1.78
cyvcf2==0.20.1 --global-option=build_ext --global-option="-I/opt/conda/envs/<ENVIRONMENT_PATH>/include/" --global-option="-L/opt/conda/envs/<ENVIRONMENT_PATH>/lib"
h5py==2.10.0
Both of these options result in disabling the use of wheel files and build from the source all the packages in requirements.txt, as described in pypa/pip#4118 (same for the --install-option argument)
And this causes a failure to install the nemo-toolkit & pysam. ( We can overcome this issue, by using an additional requirements file- one for PyPI packages that are built from source successfully need the --global-option=build_ext command argument to build against the shared libraries and another one for packages that fails and require a precompiled .whl installation. (But in my opinion, this would confuse the user more)
There was a problem hiding this comment.
Alternatively, we can take all the requirements from requirements.txt and find the corresponding packages in the Anaconda cloud, move them to the environment.yml file and leave inside the requirements.txt only the packages that do not have an Anaconda package. this way, a user that uses Anaconda will only only to use the conda env create command
…/add-required-dependencies
|
Closing for now, will reopen when prioritized |
Resolves #112
Tested with
gpuci/miniconda-cuda:11.0-devel-ubuntu20.04Our gpuCI images, which are shared among all our projects, do not install the required OS libraries through Anaconda but through the standard OS package manager. See, https://github.com/clara-parabricks/gpuci-build-environment/blob/master/Dockerfile#L32-L50. We should consider using the Anaconda-based installation steps for some combinations.